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ABSTRACT 


Large complex aerospace systems are generally validated 
in regions local to anticipated operating points rather than 
through characterization of the entire feasible operational 
envelope of the system. This is due to the large parameter 
space, and complex, highly coupled nonlinear nature of 
the different systems that contribute to the performance 
of the aerospace system. We have addressed the fac- 
tors deterring such an analysis by applying a combina- 
tion of technologies to the area of flight envelop assess- 
ment. We utilize n-factor (2,3) combinatorial parameter 
variations to limit the number of cases, but still explore 
important interactions in the parameter space in a sys- 
tematic fashion. The data generated is automatically an- 
alyzed through a combination of unsupervised learning 
using a Bayesian multivariate clustering technique (Au- 
toBayes) and supervised learning of critical parameter 
ranges using the machine-learning tool TAR3, a treat- 
ment learner. Covariance analysis with scatter plots and 
likelihood contours are used to visualize correlations be- 
tween simulation parameters and simulation results, a 
task that requires tool support, especially for large and 
complex models. We present results of simulation exper- 
iments for a cold-gas-powered hover test vehicle. 


1. INTRODUCTION 


The Modular Common Bus (MCB) project is being de- 
veloped as a family of small, low-cost spacecraft com- 
posed of modular components. The intent is that sci- 
ence instruments are solicited that meet the spacecraft’s 
mass and power budget, rather than the current practice 
of designing a custom spacecraft for each new science in- 
strument. The modular design includes components that 
enable a range of missions, including orbital and lander 
type missions. Simulink/matlab is used to rapidly pro- 
totype the design of the flight software and autogenerate 
the onboard software for the vehicle. This process also 


enhances the validation and verification of the vehicle re- 
quirements and flight software. An initial prototype been 
of the MCB lander configuration entitled the Hover Test 
Vehicle (HTV) has been developed and flown in order to 
assess this spacecraft development methodology. 

As with any complex aerospace system, the performance 
of the HTV depends on a large number of vehicle hard- 
ware parameters including center of gravity, mass, mo- 
ment of inertia, propulsion system, environment as well 
as control system parameters such as gains and dead- 
bands. A major task is to establish safe ranges for con- 
trol system parameters that ensure a safe, on-target land- 
ing given the expected variation in vehicle performance. 
Exhaustive exploration of all parameter combinations is 
infeasible for such a complex system, so, traditionally, 
parameters are randomly sampled from a defined distri- 
bution for a statistically significant number of runs (tradi- 
tional Monte Carlo testing). Vast amounts of data can be 
generated that way, and manual inspection of this data is 
usually confined to gross features of the solution (such as 
absolute compliance with requirements). Valuable trend 
and parameter sensitivities are often ignored and anoma- 
lous or unexpected data can easily be overlooked. 

Researchers have recognized that “designers must be able 
to examine various design alternatives quickly and eas- 
ily among myriad and diverse configuration possibilities ” 
(1). The number of configuration possibilities within a 
model, such as the HTV described above, can be daunt- 
ingly large. A model with only 20 binary choices already 
has 2 20 > 1, 000, 000 possible configurations, far beyond 
the capability of human comprehension. 

Part of the rapid prototyping process is to take advan- 
tage of exercising these analyses early in the design cycle. 
However the design space in this part of the life is gener- 
ally very large, greatly expanding the analysis problem. 
In these early analyses, system prototypes can interact 
with physics models in order to quickly develop usage 
policies for the software. Given the ready availability of 
super-computers, and even inexpensive LINUX clusters, 



2 


analysts may have to analyze gigabytes of data generated 
automatically from simulators. Thus, the usage policies 
need to be generated in an efficient and accesible manner 
for the analyst. 

Accordingly, since 2000 (2), we have explored sampling 
those configurations at random, running the resulting 
model, scoring the output with some oracle, then using 
data mining techniques to find the configuration options 
that most improve model output. Here, we try a new com- 
bination of methods and tools to study the configuration 
parameters on a software controller for the HTV: 

• An n-factor combinatorial test vector generation al- 
gorithm to target tests toward regions of the param- 
eter space where interactions among parameters are 
key to performance (Section 3). 

• The TAR3 minimal contrast set learner (3); is a su- 
pervised learning method that returns the minimal 
deltas between desired outcomes (hitting the target) 
and all the other outcomes (Section 4.1). 

• EM clustering algorithms that are autogenerated by 
the AutoB ayes program synthesis tool. Clustering 
is an unsupervised learning method that obtains the 
most probable estimates for class frequency and the 
governing parameters (Section 4.3). 

It was found that the combination of methods yielded 
more information than any method used in isolation. The 
operation of each algorithm can be intelligently informed 
and usefully constrained by using the output of the other. 
Combinatorial test exposes interactions between param- 
eters, TAR3 focuses the analysis on a small number of 
variables while AutoBayes reveals structures missed 
by TAR3. 

The rest of this paper discusses the problem domain and 
how it was analyzed with combinatorial test, TAR3 and 
AutoBayes. We show that data mining in combination 
with functional requirements can be used to determine 
best parameter ranges for such tasks as safely hovering 
and landing the hover test vehicle. 


2. TEST ARTICLE 


The initial test article is a prototype of a lunar landing 
configuration. The vehicle is comprised of a main pay- 
load module and propulsion module. For this earth-based 
testing, the propulsion module contains a pair of scuba 
tanks that power one main and 6 attitude control sys- 
tem thrusters. The payload module contains an inertial 
measurement unit (IMU), avionics and communications 
equipment. An image of the HTV is shown in Figure 1. 
For a complete description of the vehicle, please see ref 
(4). 

The data mining techniques discussed in this paper have 
been applied to the Hover Test Vehicle (HTV) simula- 
tion. The HTV simulation has been developed in Math- 
works Madab and Simulink (5). It contains models of 



Figure 1. Hover Test Vehicle 


the vehicle including 6DOF dynamics, propulsion sys- 
tem, effectors, sensors and avionics. Also included are 
models of the ground data station, ground station, envi- 
ronment and the flight bus. The flight software is com- 
posed of guidance, navigation, control, state estimation 
and vehicle health management models. The flight soft- 
ware portion of the simulation is autocoded using Math- 
works Real-Time Workshop Embedded Coder for control 
of the physical flight vehicle. 

The model-based development environment both speeds 
the software development process and enhances the 
workflow for validation and verification of the flight soft- 
ware to NASA standards. An extensive unit test suite 
was developed in conjunction with the models to verify 
compliance with requirements. Parametric analysis of the 
system was performed in order to assess flight readiness 
and system margins in preparation to the flight tests. 

Numerous parameter variations are considered in order 
to determine the resilience of the algorithms to disper- 
sions in mass properties, orientation and tank pressures. 
Other analyses were performed in order to assess optimal 
controller settings given the expected dispersions in ve- 
hicle configuration. Requirements on the behavior of the 
model are encoded in order to accumulate failure data, 
and all solutions are rated relative to their adherence to re- 
quirements, such as soft landing and minimum excursion 
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from the target point. The data mining techniques dis- 
cussed here are intended to characterize the operational 
envelop of the simulation and to find the ranges of param- 
eters that lead to the lengthiest flights that landing close 
to the target. The margin between the best ranges to the 
failure points of each parameter can then be determined. 

Both standard Monte Carlo and three-factor combinato- 
rial techniques were used in this study. For the Monte 
Carlo cases, one thousand cases were run, with random 
values chosen for each of the input parameters from their 
respective probability distributions. Here, the parameter 
ranges are determined such that they are likely to include 
failure cases. The output data encompass a wide range of 
variables that are saved at each iteration point. Data rep- 
resenting key parameters such as maximum altitude and 
excursion, time of flight and landing velocities were ex- 
tracted from the simulation for the analysis that follows. 
Figure 2 shows a 3D representation of 1000 simulated 
trajectories generated through a traditional Monte Carlo 
style test generation technique. 1 


Clustered Trajectories 



Figure 2. HTV trajectories for 1000 simulation runs with 
different simulation parameters. 


3. COVERING THE OPTION SPACE 


As mentioned previously, a model with 20 binary choices 
has more than a million possible configurations. For the 
ANTARES system it is anticipated that, in normal prac- 
tice, the number of parameters to vary will greatly exceed 
100, which results in an exponentially larger number of 
possible configurations. Worse yet, when dealing with 
simulations of physical systems, the input parameters are 
often real values, making choices non-discrete and the 
possible configurations infinite. So guaranteeing cover- 
age of the option space is a non-trivial problem. 


1 In this paper, colors encode the class membership as determined by 
the clustering algorithm. (Section 4.3, AutoBayes) 
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Figure 3. A 2-factor combinatorial test suite for 20 binary 
parameters 


3.1. Approach 


We have explored two approaches toward covering the 
option space, a standard Monte Carlo and a 3-factor com- 
binatorial technique. The standard Monte Carlo approach 
is simplest. It generates parameters for a simulation run 
by randomly selecting from user defined probability dis- 
tributions, such as Gaussian or uniform. The main draw- 
back from this approach is a lack of any coverage guar- 
antee, resulting in a need to run a large number of simu- 
lations to attain a given level of user confidence. Unlike 
the standard Monte Carlo approach, the combinatorial ap- 
proach makes a coverage guarantee while attempting to 
perform a minimal number of simulations. In the case of 
an n-factor combinatorial, the guarantee is that any set- 
ting of any n discrete parameters appears in at least one 
of the simulations. For instance, a 2-factor combinato- 
rial test suite for 20 binary parameters is shown in Fig- 
ure 3. Note that there are only 1 1 tests, much less than 
the million tests needed to exhaustively cover every com- 
bination. The 3-factor case only increases the number of 
tests to 26, still minuscule when compared to a million. 

While the number of tests performed using the com- 
binatorial approach is minuscule compared to exhaus- 
tive testing, anecdotal evidence suggests that this small 
number of tests is enough to catch most coding errors 
(6; 7; 8). The underlying premise behind the combinato- 
rial approach can be captured in the following four state- 
ments. 

• The simplest bugs in a program are triggered by a 
single input parameter. 

• The next simplest bugs are triggered by an interac- 
tion of two input parameters. 

• Progressively more obscure bugs involve interac- 
tions between more parameters. 

• Exhaustive testing involves trying all combinations 
of all inputs. 


So errors can be grouped into families depending on how 
many parameters need specific settings to exercise the er- 
ror. The n-factor combinatorial approach guarantees that 
all errors involving the specific setting of n or fewer pa- 
rameters will be exercised by at least one test. Applying 
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Problem Sizes 

Number of Tests 

Time 

3 4 

9 

< 1 sec 

3 13 

19 

<C 1 sec 

4 15 x 3 17 x 2 29 

35 

< 1 sec 

4 1 x 3 39 x 2 35 

29 

< 1 sec 

10 2 ° 

216 

1 sec 

3IOOO 

48 

22 sec 


Table 1. Performance of test suite generator on 2-factor 
combinatorial problems 

an n-factor combinatorial approach to testing ANTARES 
involved characterizing each real- valued parameter as a 
partition of discrete ranges. When turning a computed 
test into a simulation run, each range is replaced by a 
real value chosen from a uniform distribution across that 
range. The result is a multidimensional space of simula- 
tion runs that projects down to a uniform distribution on 
any plane of input parameters. 


of the hypercube at 1, 1, ... is the apex of the cube and 
represents the desired goal for the system. All the exam- 
ples are scored by their normalized Euclidean distance to 
the apex. 

For this study, outputs were scored on one dimension- 
distance to the desired target. Future studies will exploit 
more of BORE’s facilities and will score outputs on other 
dimensions such as average standard deviation on attitude 
control, minimum use of propellant, minimal maximum 
G-force, etc. 

For each run i of the simulator, the n outputs 2Q are nor- 
malized to the range 0 ... 1 as follows: 

7 \j Xj—Tnin(X) 

* max(X)—min(X) 

The Euclidean distance of {AT 1? AT 2 , ...} to the ideal po- 
sition of {Ni = 1,AT 2 = 2,...} is then computed and 
normalized to the range 0..1 as follows: 


3.2. Implementation 


Wi = 1- 


y/N? + gf + - 

\/n 


To generate 2-factor combinatorial test suites there are a 
number of algorithms in the literature (9). Our algorithm 
is a generalization of IPO (10) to facilitate generating n- 
factor combinatorial test suites in addition to a number of 
features that a real world test suite generator would need 
(11). These features include the ability to explicitly in- 
clude particular combinations, explicitly exclude partic- 
ular combinations, require n-factor combinatorial cover- 
age of specific subsets of parameters, and tie the existence 
of particular parameters to the setting of another parame- 
ter. 

The resulting algorithm is 1041 lines of documented Java 
code with an example output that appears in Figure 3. 
Even with these extra capabilities the algorithm generates 
test suites that are comparable to those generated by the 
more restricted systems in the literature. As shown in 
Table 1 the code generates quality solutions very rapidly 
on a 400MHz Windows laptop. In the problem sizes, the 
X Y syntax means that there are Y X- valued parameters. 


4. DATA ANALYSIS TOOLS & METHODS 


4.1. TAR3 

Multi-Dimensional Optimization. BORE, short for 
best or rest , takes instances scored on multiple utilities as 
input and classifies each of them “best” or “rest”. BORE 
maps the instance outputs into a hypercube, which has 
one dimension for each utility. 


where higher W{ (0 < W{ < 1) correspond to better 
runs. This means that the W{ can only be improved by 
increasing all of the utilities. To determine the “best” and 
“rest” values, all the W{ scores were sorted according to 
a given threshold BEST. The top BEST% are then classi- 
fied as “best” and the remainder as “rest”. 


Treatment Learning with TAR3. Once BORE has clas- 
sified the data into best and rest , a data miner is used 
to find input settings that select for the better outputs. 
This study uses the TAR3 data miner since this learn- 
ing method returns the smallest theories that most effect 
the output. This means that TAR3 tries to determine a 
minimal set of model parameters, which have the most 
influence on the behavior of the simulation system. 


TAR3 inputs a set of training examples E. Each example 
maps a set of attribute ranges to some class symbol; i.e., 
{i^, Rj,. . . — ► C}. The class symbols Ci, C 2 , . . . are 
stamped with some utility score that ranks the classes; 
i.e., {Ui < U 2 < ... < Uc}- With E , these classes 
occur at frequencies F 2 , . . . , F c where E — 1* 
A “treatment” T of size M is a conjunction of attribute 
ranges {#1 AR^ . . • AR m }• Some subset of e C E is con- 
sistent with the treatment. In that subset, the classes oc- 
cur at frequencies / 1 , / 2 , • • ■ , fc- TAR3 seeks the small- 
est treatment T which induces the biggest changes in 
the weighted sum of the utilities times frequencies of the 
classes. Formally, this is called the lift of a treatment: 


lift = 


Ec u cfc 

E cUcFc 


BORE normalizes instances scored on N dimensions Classes in treatment learning get a score Uc and the 

into “zero” for “worst”, and “one” for “best”. The comer learner uses this to assess the class frequencies resulting 
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from applying a treatment (i.e., applying constraints to 
the inputs). In normal operation, a treatment learner does 
controller learning that finds a treatment, which selects 
for better classes and reject worse classes By reversing 
the scoring function, treatment learning can also select 
for the worse classes and reject the better classes. This 
mode is called monitor learning since it finds the thing 
we should most watch for. 

Formally, treatment learning is a weighted-class mini- 
mal contrast-set association rule learner. The treatments 
are associations that occur with preferred classes. These 
treatments serve to contrast undesirable situations with 
desirable situation where more of the outcomes are fa- 
vorable. Treatment learning is different to other contrast 
set learners like STUCCO (12) since those other learners 
don’t focus on minimal theories. 

Conceptually, a treatment learner explores all possible 
subsets of the attribute ranges looking for good treat- 
ments. Such a search is infeasible in practice so the 
art of treatment learning is quickly pruning unpromis- 
ing attribute ranges. This study uses the TAR3 treatment 
learner (13) that uses stochastic search to find its treat- 
ments. 


4.2. TAR3: Results 

When applied to our spacecraft re-entry simulator, 
BORE/TAR3 works as follows. Firstly, BORE gener- 
ates a baseline distribution where the simulation outputs 
are divided into two categories: 25% are classified as best 
(closest to the target region) while the others were rest 
(the remaining 75% of the examples). The first row in 
Table 2 shows the initial setting; the worth of this sce- 
nario is defined to be one. 


Tr 

worth 

Constraint 

best 

rest 

- 

1.0 

- 

25% 

75% 

1 

1.46 

0 < c < 0.34 A 
0.68 < / < 1.0 

82% 

18% 

2 

1.43 

0.67 < c < 1.0 A 
0 < / < 0.34 

79% 

21% 

3 

1.15 

0.85 < c < 0.98 A 
0.67 < / < 1.0 

44% 

56% 


Table 2. TAR3 treatments 

TAR3 then seeks the minimal delta between best and 
rest. Several candidate deltas are generated and scored 
according to their normalized worth with respect to the 
baseline. The top three deltas as produced by TAR3 are 
shown in Table 2. Each treatment consists of a conjunc- 
tion of linear constraints on the input variables. For in- 
puts adhering to the constraints of the treatment, their 
effect is calculated. In this case, a restriction of the in- 
put variables c and / (two of 24 simulation parameters) 
to the shown ranges causes substantial improvement over 
the initial setting, as the “best” and “rest” values, which 


BORE produces, show. The relative improvement is in- 
dicated by the worth of the treatment. 

Note that: 

• The deltas with the highest worth have the greatest 
percentage of best examples. 

• The top two treatments have very similar worths, 
and the third much less so. 

• The top two treatments comment on different ex- 
tremes of the variables c, /. 

• Only very few of the 24 variables in the simulation 
appear in the treatments. 

Of these effects, the last is most important. TAR3 is a fo- 
cus tool that tells an analysts “of all the things you might 
think about, these few variables are all you should be 
considering”. 


4.3. AutoRayes 

A well-known method to find structure in large sets 
of data is to perform clustering. Clustering is an un- 
supervised learning method that tries to estimate class 
membership matrix and class parameters, only given 
the data. A variety of algorithms can be used, for 
example, the EM-algorithm (14). A number of EM- 
implementations is available (e.g., Autoclass (15), EM- 
MIX (16), or MCLUST (17)) and could be used for this 
problem. 

However, in order to refine the statistical model (e.g., 
by incorporating other probability distributions for cer- 
tain variables or to introduce domain knowledge), the 
EM-algorithm needs to be modified substantially for 
each problem variant, making experimentation a time- 
consuming and error-prone undertaking. Thus, we are 
using AutoBayes tool to produce customized variants 
of the EM-algorithm. 

AutoBayes (18) is a fully automatic program synthesis 
system that generates efficient and documented C/C++ 
code from abstract statistical model specifications. De- 
veloped at NASA Ames, AutoBayes is implemented in 
approximately 90,000 lines of documented SWI Prolog 
code. From the outside, it looks similar to a compiler for 
a very high-level programming language: it takes an ab- 
stract problem specification in the form of a (Bayesian) 
statistical model and translates it into executable C/C++ 
code that processes the data or, in our case, can be called 
from Matlab. 

On the inside, however, it works quite different: 
AutoBayes first derives a customized algorithm skele- 
ton implementing the model and then transforms it into 
optimized C/C++ code. The input specification is trans- 
lated into a Bayesian Network (19), which is a compact 
internal representation of the statistical model. Then, 
the program synthesis system uses a schema based ap- 
proach to translate the statistical problem into smaller 
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model mog as 'Multivar. Mix of Gaussians'; 

const int D as 'number of variables' 
const int N as 'number of data points' 
const int C as 'number of classes' 
with 1 < C; with C << N; 

double phi(l..C) as 'class probabilities' 
with 1 = sum(_i := 1..C, phi(_i)); 
double mu (1 . . D, 1..C), sigma(l..D, 1..C); 

output int c(l..N) as 'latent variable'; 
c (_) “ discrete (phi) ; 

data double x(l..D, 1..N); 

x(_i,_j) “ gauss (mu (_i,c(_j) ), sigma (_i,c(_j) )) ; 
max pr(x|{phi, mu, sigma}) wrt {phi, mu, sigma}; 

Figure 4. AutoBayes specification for mixture model 

problems and then, after symbolically solving subprob- 
lems, to transform the instantiated customized algorithm 
into efficient code. This task is heavily supported by a 
domain-specific schema library, an elaborate symbolic 
subsystem, and an efficient rewriting engine. After op- 
timization C or C++ code is generated for various plat- 
forms (e.g., embedded systems, Matlab, Simulink, or Oc- 
tave). For our experiment, we used AutoBayes to gener- 
ate code that can be called from Matlab as a MEX func- 
tion. 

The basic statistical model used for this study describes 
the properties of the data in a fully declarative fashion: 
for each problem variable of interest (i.e., observation 
or parameter), properties and dependencies are specified 
via probability distributions and constraints. Figure 4 
shows how our clustering, a Gaussian mixture model 
with diagonal covariance matrices can be represented in 
AutoBayes’s specification language. The model as- 
sumes that the data consists of N points in D dimen- 
sions such that each point belongs to one of C classes; 
the first few lines of the specification just declare these 
symbolic constants and specify the constraints on them. 
Each point x ( 1 . . C, _ j ) (where . . corresponds to 
Matlab’s subrange operator : , and _i, _ j are index vari- 
ables) is drawn independently from a univariate Gaus- 
sian with mean mu (_i, c (_ j ) ) and standard deviation 
sigma (_i, c (_j ) ) . The unknown distribution param- 
eters can be different for each class and each dimension; 
hence, we declare them as matrices. The unknown as- 
signment of the points to the distributions (i.e., classes) 
is represented by the latent variable c; since we are in- 
terested in the classification results as well (and not only 
the distribution parameters), c is declared as output, c 
is distributed as a discrete distribution with the relative 
class frequencies given by the also unknown vector phi. 
Since each point must belong to a class, the sum of the 
probabilities must be equal to one. Finally, we specify 
the goal inference task, maximizing the conditional prob- 
ability pr (x | {phi, mu, sigma}) with respect to the 
parameters of interest, phi, mu, and sigma. This means 
that we are interested in a maximum likelihood estimate 
(MLE) of the model parameters. 

If additional domain knowledge, e.g., priors on the mean 
values of the features for each class are known, more 


const double mu_0(l..D, 1..C) as 'expected means' 
const double kappa_0 (1 . . D, 1..C) as 'confidence' 
with 1 < kappa_0 (_,_) ; 
const double sigma_0 (1 . . D, 1..C); 
double mu(l..D, 1..C); 

mu (_i,_j) ~ gauss (mu_0 ( i , j ) , 

sigma ( i , j ) *kappa_0 (_i,_j) ; 

max pr ({mu, sigma, x}|phi) wrt {phi, mu, sigma}; 

Figure 5. Additions to AutoBayes specification for 
Gaussian mixture model with priors ( additional domain 
knowledge) 

complicated models (e.g., maximum aposteriori esti- 
mates, MAP) can be easily specified. Only a few lines 
(Figure 5) with specification of the prior and an updated 
maximization goal is necessary to produce a substantially 
different data analysis algorithm. Note that all these mod- 
els are completely declarative and do not require the user 
to prescribe any algorithmic aspects of the estimation 
program. 


5. RESULTS 


The data produced by the simulation are actually time se- 
ries data over a large number of variables. Characteristic 
parameters were extracted from the data stream and uti- 
lized in this analysis. Data dimensions obviously include 
the landing position, the sum of consumed fuel, maximal 
structural loads, as well as a measure of the duration of 
extended time intervals where the gravitational forces ex- 
ceed a safe limit. With this preprocessing, we obtained a 
data set with 10 dimensions. All data were normalized. 

These data then were clustered using the Madab/C code 
as was generated by AutoBayes (790 lines of docu- 
mented C code). The generated data analysis algorithm 
determined that with 10 classes, a good separation can be 
achieved. 

The results of clustering, relative to time of flight and ini- 
tial wet mass is shown in Figure 6. Different colors indi- 
cate into which class a specific simulation run falls. The 
classes are ranked using a penalty function based on land- 
ing velocity and position error. Blue indicates the lowest 
penalty function, while red indicates the highest penalty 
function. The trend seen in the figure is that lower times 
of flight generally have lower penalty functions. Wet 
mass is key to time of flight, but other factors contribute 
to the failure clusters exhibited in the plot. Here, the yel- 
low through red classes show repeated violations of the 
landing requirements. Overall statistics for the compli- 
ance with performance requirements was: 

• Maximum Position Excursion no greater than 3 M: 
258 cases failed 

• Vertical Velocity on Landing no greater than 4.0 
M/S: 167 failed 



7 


• Horizontal velocity on Landing no greater than 1.0 
M/S: 49 failed 

Some of the cases failed more than one of the require- 
ments, so the total number of cases without failure was 
656. This simulation exhibited a sufficient range of ac- 
ceptable/failure cases such that the key parameter values 
defining the failure front chould be determined. 



Figure 6. Relationship between Time of Flight and Initial 
Wet Mass Colors indicate class membership. 

To determine the root cause of the poor landing clusters, 
TAR3 was utilized to determine the key parameters driv- 
ing the simulation behavior. It identified several key pa- 
rameters, that in conjunction with each other drove the 
failure behavior. In particular, TAR3 identified the fol- 
lowing combined set of parameter ranges that induced 
failure: 

• MAS cgy location wet=0.005951.. 0.007885 

• MAS Iyz wet=-0.961 177. .-0.835669 

• MAS wet=68. 826 1 03 . .69.489502 

• INI rotx=0.928976.. 2.996930 

This indicates that the initial mass, y center of gravity in 
combination with one the Iyz moment of inertia and the 
inital x rotation of the vehicle would cause unacceptible 
behavior. This finding introduced a set of day-of-flight 
flight rule restrictions on initial orientation and a more 
careful evaluation of the center of gravity of the vehicle. 
It was determined that reduced mass in the tanks would 
lead to a lower y-cg offset, so a reduced tank pressure was 
also utilized for the first free-hover test flight. 

Figure 7 shows contours of “likelihood” for two parame- 
ters identified as critical by TAR3. To generate this plot, 
the input domain was discretized and the likelihood of 
success was computed for each cell. Here, Likelihood 
combines the overall probability of success in the whole 


population, ratio of success in the local cell, and local 
cell population. Likelihood will approach one in well- 
populated cells with a high ratio of success, and will ap- 
proach zero if either there is poor statistical support or a 
low ratio of success. If a variable is not correlated, then 
the local likelihood will approach the overall ratio of suc- 
cess in the complete population. Using this metric, it is 
seen that choosing a lower mass and carefully orienting 
the vehicle improves the resiliance of the simulation to 
dispersions in other key parameters. 



MAS wet 


Figure 7. Likelihood of successful flight relative to initial 
x rotation and mass 


6. CONCLUSIONS 


In this paper, we have explored a combination two learn- 
ers (AutoBayes and TAR3) to explore the internal state 
space of some flight guidance software with combinato- 
rial test techniques. The combination of these technolo- 
gies revealed features that would have been invisible for 
state of the practice. Further, the experiment suggests 
some novel ways that these technologies could usefully 
augment each other. 

In the case of the first test flight of the hover test vehi- 
cle, the combined technologies provided guidance as to 
strategies that would increase the chances of a successful 
test flight. In fact, the hover test vehicle performed in the 
manner suggested by the simulation, and a successful set 
of free-flying test flights were conducted. 

More generally, given the growing importance of model- 
based reasoning in software engineering, the ability to 
use data miners to find and constrain the most important 
parts of our software models, should prove to be a tech- 
nique of growing importance in the years to come. 
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